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ABSTRACT 

Recent  requirements  to  establish  the  credibility  of  the  models  and  simulations  (M&S)  that  are 
used  in  Department  of  Defense  (DoD)  applications  have  resulted  in  the  development  of  several 
approaches  to  M&S  credibility  assessment.  Each  of  these  approaches  attempts  to  integrate 
verification  and  validation  (V&V)  activities  into  a  formal  process  whose  products  (hopefully) 
support  M&S  accreditation.  The  cost  of  "VV&A"  activities  has  not  typically  been  a  driving 
factor  in  the  development  of  these  processes,  however,  and  their  independent  and  fragmentary 
development  has  tended  to  result  in  "point-solution"  approaches  to  the  problem  of  cost- 
effectiveness.  Despite  their  diversity,  however,  many  VV&A  processes  share  features  in 
common,  suggesting  that  it  may  be  possible  to  identify  the  essential  elements  of  cost-effective 
VV&A. 

This  paper  proposes  five  "common  sense"  elements  of  cost-effective  VV&A,  and  describes  how 
each  of  these  elements  can  contribute  to  the  cost-effectiveness  of  VV&A  processes  in  general.  A 
comparative  analysis  of  six  VV&A  processes  currently  in  use  to  support  M&S  accreditation  for  a 
wide  variety  of  DoD  applications  is  then  presented.  The  analysis  reveals  that  the  five  elements 
proposed  as  essential  to  cost-effective  VV&A  do,  in  fact,  characterize,  in  whole  or  in  substantial 
part,  all  of  the  VV&A  processes  considered.  The  paper  concludes  by  considering  some  of  the 
evaluative  implications  of  these  findings. 


INTRODUCTION 

Various  elements  of  the  M&S  community  have  attempted  to  identify  and  define  the  technical  and 
formal  requirements  for  the  accreditation  of  particular  M&S  types  or  particular  classes  of  M&S 
applications,  but  there  remains  considerable  concern  over  the  cost  of  VV&A  activities.  Recent 
attempts  at  addressing  the  cost  issue  have  focused  primarily  on  reducing  the  number  of  dollars 
spent  on  VV&A,  rather  than  on  improving  the  "effectiveness"  of  those  dollars  that  are  spent.  An 
optimum  cost  solution  to  the  VV&A  problem,  however,  must  take  into  account  two  factors:  1)  an 
objectively  justifiable  reduction  in  the  number  of  activities  that  are  performed  (and  thus  the 
dollars  spent),  and  2)  a  logical  selection  of  the  most  appropriate  type  of  activities  to  perform, 
based  on  their  contributions  to  M&S  credibility  relative  to  their  typical  cost.  The  challenge  is  to 
define  what  features  of  VV&A  processes  need  to  be  present  in  order  for  optimum  cost  solutions 
to  be  feasible.  This  paper  attempts  to  identify  the  elements  of  cost-effective  VV&A,  and  to 
generalize  these  elements  well  enough  for  widespread  application. 
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ELEMENTS  OF  COST-EFFECTIVENESS 


In  trying  to  define  a  robust  set  of  factors  that  can  be  used  to  evaluate  and  improve  the  cost- 
effectiveness  of  VV&A  processes,  we  need  to  make  sure  that  the  factors  proposed  are  generic 
enough  to  be  broadly  applicable,  and  yet  substantive  enough  to  be  technically  useful  in  practical 
application.  To  develop  a  logical  set  of  generic  requirements  for  cost-effective  VV&A,  we  might 
reason  as  follows: 

"It  stands  to  reason  that  cost  savings  in,  and  improved  effectiveness  of,  VV&A  must  result  from 
at  least  two  factors:  skillful  focusing  and  tailoring  of  VV&A  activities  to  meet  objectively 
defined  M&S  requirements  for  specific  applications;  and  reuse  of  prior  VV&A  information 
wherever  possible.  By  analyzing  the  application  at  hand,  and  focusing  and  tailoring  VV&A 
activities  to  meet  objectively  defined  credibility  requirements  for  any  M&S  that  might  be 
considered  for  use  in  the  application,  one  can  prevent  superfluous  activities  from  being 
conducted;  and  by  reusing  prior  VV&A  information  wherever  possible,  one  can  avoid  paying  for 
information  that  is  already  available. 

"But  'focusing  and  tailoring'  of  VV&A  activities  would  seem  to  require  that  there  be  some 
recognized  and  reliable  way  to  choose  the  most  appropriate  activities  from  among  those  possible 
to  perform,  and,  moreover,  to  tailor  these  activities  to  maximize  relevance  to  the  application  at 
hand.  In  order  for  such  a  tailoring  process  to  yield  reliable  cost  savings  and  improved 
effectiveness,  while  still  meeting  the  requirements  of  the  application,  two  further  prerequisites 
seem  to  be  implied:  the  range  of  applications  to  which  the  tailoring  process  applies  would  have 
to  be  well-defined,  as  would  some  standard  list  of  applicable  VV&A  activities,  each  of  which  is 
known  to  add  well-defined  and  unique  elements  of  credibility  to  M&S. 

"Effective  'reuse  of  VV&A  information'  would  seem  to  require  that  VV&A  techniques  and 
procedures  be  defined  and  standardized  for  application  to  particular  M&S  types  or  to  particular 
classes  of  M&S  application;  otherwise,  there  would  be  no  way  to  ensure  consistency  of 
interpretation  of  VV&A  results  across  the  spectrum  of  possible  M&S  types  and  M&S 
applications.  Since  no  one  would  be  sure  just  what  substantive  information  a  particular  type  of 
VV&A  report  might  contain,  the  attempted  reuse  of  VV&A  information  would  become  too 
cumbersome  to  be  effective.  In  addition,  VV&A  results  would  have  to  be  reported  in  a 
standardized  way,  to  minimize  confusion,  and  to  form  the  basis  for  an  accumulation  of  evidence 
of  M&S  credibility  over  time.  Finally,  easy  access  to  any  prior  VV&A  information  generated  by 
the  standard  process  and  documented  in  the  standard  fashion  would  be  essential  to  widespread 
reuse  of  standardized  VV&A  data  sets  for  different  types  of  M&S  or  different  classes  of  M&S 
application." 

If  we  summarize  the  requirements  for  cost-effective  VV&A  implicit  in  the  above,  we  have: 

1.  A  well-defined  menu  of  VV&A  activities,  each  of  which  provides  a  well-defined,  unique 
(and  community-recognized)  contribution  to  M&S  credibility; 

2.  A  process  for  selecting  VV&A  activities  from  this  menu  based  on  requirements  derived 
from  detailed  analysis  of  individual  applications; 


3.  A  well-defined  (and  documented)  process  for  performing  each  of  the  VV&A  activities; 

4.  A  set  of  standards  and  guidelines  for  reporting  (and  accumulating)  VV&A  results,  and; 

5.  A  repository  of  readily  accessible  and  easily  retrievable  prior  VV&A  results. 

The  following  sections  describe  these  "requirements"  in  detail,  and  discuss  their  potential 
contribution  to  the  cost-effectiveness  of  VV&A. 

Menu  of  Activities 

A  VV&A  menu  is  a  list  of  activities,  each  of  which  produces  explicitly  defined  technical 
information,  and  each  of  which  contributes  to  M&S  credibility  in  a  unique  way.  Explicit 
definitions  prevent  confusion  as  to  what  is  meant  by  each  activity;  a  requirement  for  each 
activity  to  contribute  uniquely  to  M&S  credibility  reduces  technical  overlap  between  ostensibly 
similar  activities  (or  their  products),  as  well  as  the  tendency  to  duplication  of  effort. 

A  VV&A  menu  should  consist  of  a  list  of  possible  activities,  a  concise  technical  description  of 
each  activity  (including  what  tasks  are  required  to  complete  that  activity),  a  general  description 
of  the  documentation  product  that  should  result  from  the  activity,  and  a  clear  identification  how 
the  given  activity  contributes  to  M&S  credibility.  Insofar  as  possible,  VV&A  activities  should  be 
categorized  and  subdivided  according  to  the  type  of  information  they  provide,  and  groups  of 
activities  that  form  coherent  subsets  of  critical  information  should  be  identified. 

Since  no  single  set  of  activities  will  be  universally  applicable  to  every  type  of  M&S  or  to  every 
class  of  M&S  application,  VV&A  menus  will  tend  to  be  focused  on  the  unique  requirements  of 
the  M&S  communities  for  which  they  are  developed.  Tailored  VV&A  menus  can  help  convert 
VV&A  from  a  "black  art",  in  which  the  technical  definition  and  implementation  of  each  activity 
is  in  the  eye  of  the  beholder,  into  a  more  cost-efficient  and  repeatable  "science",  in  which 
modifications  to  basic  techniques,  which  might  be  necessitated  by  the  specifics  of  particular 
applications,  proceed  from  a  stable  technical  baseline. 

Tailoring  Process 

The  importance  of  guidelines  to  assist  M&S  users  in  the  task  of  optimizing  VV&A  activities  and 
reducing  VV&A  costs  cannot  be  overstated.  Without  a  standard  means  of  tailoring  VV&A 
activities  to  the  requirements  of  individual  M&S  or  M&S  applications,  there  will  be  a  tendency 
to  perform  every  activity  on  the  menu,  in  the  hopes  of  "proving"  (through  sheer  volume, 
presumably)  that  the  subject  M&S  have  no  discernible  deficiencies.  The  "good"  becomes  the 
enemy  of  the  "best",  leading  to  the  performance  of  unnecessary  or  objectively  unjustifiable 
VV&A  activities.  The  end  result  is  a  reinforcement  of  the  perception  (prevalent  today)  that 
VV&A  "costs  too  much  and  takes  too  long",  as  well  as  the  dilution  of  objective  requirements  for 
accreditation  simply  to  meet  a  fixed  (usually  meager)  VV&A  budget.  If  an  objective  basis  upon 
which  to  establish  M&S  credibility  is  a  goal,  a  process  to  select  and  tailor  VV&A  activities  in  a 
way  that  meets  objectively  defined  accreditation  requirements  at  minimum  cost  is  essential. 


MENU  OF  ACTIVITIES 
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Figure  1:  Summary  of  Prerequisites  to  Cost-Effective  VV&A 
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To  enhance  cost-effectiveness,  the  VV&A  task  selection  and  tailoring  process  should  address  the 
tailoring  problem  from  two  points  of  view.  The  first  aspect  of  the  tailoring  process  should 
provide  a  means  of  identifying  and  prioritizing  the  M&S  functional  and  fidelity  requirements  of 
the  application,  and  relating  these  requirements  to  the  most  appropriate  set  of  V&V  activities, 
regardless  of  cost.  The  second  aspect  of  the  tailoring  process  should  assign  weights  related  to 
informative  "value"  to  each  proposed  activity  based  on  historical  (or  anecdotal)  data,  and  should 
result  in  that  mix  of  VV&A  activities  for  each  required  M&S  function  that  maximizes  credibility 
while  minimizing  cost.  It  is  interesting  to  note  that  these  "weights"  can  be  more  easily  assigned 
once  the  substantive  contribution  of  each  activity  is  well  defined  and  understood,  as  above. 

Documented  Procedures 

A  cost-effective  VV&A  process  should  include  detailed,  documented  procedures  for  performing 
each  of  the  tasks  contained  in  the  VV&A  menu.  Well-defined  and  documented  procedures  are 
important  for  three  reasons:  first,  they  contribute  to  a  better  understanding  of  the  purpose  of  each 
activity,  reducing  confusion  and  encouraging  consensus  about  the  value  of  each;  second,  they  aid 
the  M&S  user  in  planning  and  executing  a  tailored  VV&A  effort  by  forming  a  sound  technical 
basis  for  excursions  from  the  task  baseline,  as  well  by  allowing  a  history  of  cost  and  schedule 
data  associated  with  each  activity  to  be  developed,  and;  third,  they  improve  the  efficiency  of  task 
execution  by  reducing  guesswork,  focusing  technical  questions  and  developing  a  detailed  and 
nuanced  corporate  memory  and  expertise  in  each  task  area. 

VV&A  activities  should  be  integrated  with  appropriate  aspects  of  the  M&S  development  and 
configuration  management  processes  wherever  possible,  so  that  M&S  capability  does  not  get  out 


of  step  with  M&S  credibility.  Integration  of  VV&A  with  M&S  development  and  configuration 
management  allows  VV&A  information  to  be  related  to  specific  M&S  versions,  and  establishes  a 
VV&A  "pedigree"  that  can  be  used  to  substantiate  the  credibility  of  unchanged  portions  of  the 
M&S  as  later  versions  are  developed. 

Documents  that  describe  VV&A  activities  should  lay  out  VV&A  procedures  and  techniques  in  a 
logical,  incremental  sequence  wherever  possible,  so  that  the  products  of  earlier  activities  can  be 
used  to  accomplish  later  activities,  reducing  duplication  of  effort.  This  is  especially  important 
where  multiple  and  diverse  participants  in  the  VV&A  effort  may  be  involved. 

Documentation  Standards 

Cost  avoidance  in  VV&A  is  directly  dependent  on  the  availability  (and  utility)  of  prior  results. 
Without  the  ability  to  reuse  VV&A  information,  each  M&S  user  is  forced  to  perform  VV&A 
independently,  resulting  in  a  strong  tendency  toward  duplicative  efforts.  The  usefulness  of  prior 
VV&A  information,  however,  depends,  in  large  part,  on  documentation  standards  that  reduce  or 
eliminate  the  problem  of  having  to  "interpret"  the  results  of  VV&A  activities  performed  and 
reported  by  others  in  different  ways.  Moreover,  documentation  standards  form  a  convenient  basis 
for  the  accumulation  of  VV&A  information  over  time. 

The  term  "standards"  refers  to  standardized  information  content  and  organization,  not  printing 
guidelines  (e.g.,  margin  size,  font  type  and  size,  footnote  and  heading  conventions,  etc.), 
although  these  latter  can  certainly  help  the  readability  and  ease  of  use  of  VV&A  documentation. 
Some  examples  of  standard  types  of  VV&A  information  might  be: 

•  Identification  of  the  M&S  version  on  which  the  VV&A  activity  was  performed 

•  Description  of  the  VV&A  activity  itself,  especially  any  excursions  from  the  accepted 
technical  baseline  definition  of  the  activity 

•  Description  of  the  results  of  the  VV&A  activity 

•  ID  of  any  M&S  limitations  that  follow  from  the  results  of  the  particular  VV&A  activity 

•  Impacts  on  M&S  usage  for  the  specific  application  that  are  implied  by  the  results  of  the 
VV&A  activity 

Data  Repository 

Documentation  standards  are  insufficient,  by  themselves,  to  allow  reuse  of  VV&A  information 
on  a  wide  scale.  VV&A  results  must  also  be  readily  accessible  to  M&S  users.  This  means  that 
VV&A  information  must  be  archived  and  indexed  in  such  a  way  that  it  can  be  easily  located, 
extracted  and  distributed  to  typical  M&S  users.  A  database  of  summary  results  explicitly  linked 
to  the  activities  contained  in  the  VV&A  menu  can  also  serve  as  a  powerful  M&S  selection  tool 
in  cases  where  multiple  M&S  are  under  consideration  for  use  in  a  given  application.  Such  an 
archive  can  be  used  to  filter  M&S  from  among  competing  alternatives  based  on  their  defined 
functional  capabilities  and  VV&A  histories,  and  to  identify  VV&A  activities  already  performed. 
This  reduces  the  tendency  to  repeat  activities,  and  can  serve  to  focus  the  M&S  community's 
attention  on  gaps  in  the  composite  VV&A  record.  As  VV&A  data  gaps  are  filled  over  time  by 


M&S  users,  the  entire  M&S  community  benefits  from  the  use  of  common  VV&A  data  resources. 


PREVALENCE  OF  THE  ELEMENTS 

At  this  point,  one  might  reasonably  ask  for  evidence  that  the  elements  proposed  as  essential  to 
cost-effective  VV&A  comprise  a  robust  (and  relevant)  listing.  To  answer  this  question,  we 
conducted  a  review  of  six  major  VV&A  processes  across  DoD  in  the  hopes  determining  whether 
or  not  any  of  the  proposed  elements  of  cost-effectiveness  were  widely  used  in  actual  practice. 
We  anticipated  three  possible  outcomes  from  such  a  review: 

Outcome  1:  The  proposed  features  of  cost-effective  VV&A  would  be  evident  (in  some  form)  in 
all  or  most  of  the  existing  VV&A  processes  examined.  In  this  case,  since  the  processes  chosen 
for  analysis  were  developed  independently  for  a  wide  variety  of  M&S  types  and  M&S 
applications,  we  would  conclude  that  the  features  proposed  as  essential  to  cost-effective  VV&A 
are  somehow  truly  essential,  or  normative. 

Outcome  2:  None  (or  very  few)  of  the  proposed  features  of  cost-effective  VV&A  would  be 
found  (in  any  form)  in  any  of  the  VV&A  processes  examined,  but  other  features  of  these  VV&A 
processes  might  be  similar  to  each  other,  and  be  related  to  cost-effectiveness.  In  this  case  we 
would  conclude  that  we  were  barking  up  the  wrong  tree  with  our  prescription  for  cost-effective 
VV&A,  and  that  the  existence  of  common  features  of  VV&A  processes  other  than  we  had 
proposed  as  essential  to  cost-effectiveness  would  imply  that  these  (new)  features  should  be  the 
ones  abstracted  and  generalized  for  use  by  the  larger  M&S  community. 

Outcome  3:  There  would  be  no  correlation  between  our  proposed  features  of  cost-effective 
VV&A  and  actual  VV&A  processes,  and  no  truly  common  features  among  the  VV&A  processes 
themselves.  In  this  case,  we  would  conclude  that  VV&A  process  development  might  be  too 
closely  linked  to  the  type  of  M&S  or  M&S  application  to  result  in  common  features  that  could  be 
generically  related  to  the  cost-effectiveness  of  VV&A. 

Six  VV&A  processes  were  the  subject  of  comparison  and  analysis,  each  of  which  is  widely  used 
(or  accepted  as  normative)  within  the  DoD  M&S  community  that  it  serves: 

•  The  VV&A  process  used  by  the  Aggregate  Level  Simulation  Protocol  (ALSP)  Program 
for  use  on  the  Joint  Training  Confederation  (JTC). 

•  The  Analyst  Tool  Box  (ATB)  M&S  assessment  process  developed  by  the  Ballistic 
Missile  Defense  Organization  (BMDO). 

•  The  9-Step  VV&A  Process  used  by  the  Distributed  Interactive  Simulation  (DIS) 
Program. 

•  The  VV&A  process  used  by  the  Joint  Modeling  and  Simulation  System  (J-MASS)  for 
component  threat  simulations. 

•  The  VV&A  process  developed  by  the  Military  Operations  Research  Society  (MORS) 
during  its  Simulation  Validation  (SIMVAL)  Workshop  series. 


•  The  VV&A  process  developed  by  the  Susceptibility  Model  Assessment  with  Range  Test 
(SMART)  Project  for  M&S  used  in  airborne  weapon  system  acquisition  and  testing. 
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Table  1:  Matrix  of  V&V  Processes  and  Attributes 


The  results  of  the  analysis  are  shown  in  table  1.  In  this  table  the  letter  "D"  indicates  that  a 
requirement  for  the  specified  VV&A  process  feature  is  formally  documented,  but  not 
implemented  (for  a  variety  of  reasons)  in  practice.  (See  the  authors'  paper  mentioned  in  footnote 
reference  6  for  more  details).  The  letter  "I"  indicates  that  the  process  feature  is  implicitly  or 
explicitly  implemented  in  the  VV&A  process  considered. 

Table  1  shows  that  two  of  the  five  features  proposed  as  essential  to  cost-effective  VV&A  are 
exhibited  by  all  of  the  processes  examined.  These  are: 

•  A  well-defined  menu  of  VV&A  activities,  each  of  which  has  a  clearly  defined  and  well- 
accepted  relationship  to  M&S  credibility,  and; 

•  A  requirement  for  (or  the  existence  of)  a  repository  of  prior  VV&A  data  that  is  readily 
accessible  to  M&S  users. 

Two  of  the  features  proposed  as  essential  to  cost-effective  VV&A  were  exhibited  by  five  of  the 
six  VV&A  processes  examined.  These  are: 

•  A  VV&A  activity  selection  and  tailoring  process  that  translates  M&S  application 
requirements  into  VV&A  requirements,  and; 

•  A  set  of  documented  procedures  for  VV&A  techniques  and  procedures  that  provides  both 
technical  details  and  a  discussion  of  expected  product. 

The  feature  "common  reporting  standards"  was  addressed  only  by  the  MORS  and  SMART 
VV&A  processes.  A  recent  review  of  the  ALSP  methodology,  however,  resulted  in  a 
recommendation  that  common  reporting  standards  be  adopted.  This  recommendation  was 
favorably  received  by  the  ALSP  Program  Office  and  may  be  reflected  in  future  Accreditation 
Reports  produced  by  that  program.  The  development  of  the  DIS  9-Step  VV&A  process  did  not 


originally  consider  the  benefits  of  common  reporting  standards.  Recent  interactions  with  the 
SMART  Project  via  the  Defense  Modeling  and  Simulation  Office  (DMSO),  however,  have 
resulted  in  endorsement  at  least  of  the  concept,  as  well  as  discussions  related  to  incorporating  it 
into  the  DIS  approach  to  VV&A. 


CONCLUSIONS 

We  admit  that  it  has  not  been  established,  either  logically  or  empirically,  that  the  five  attributes 
proposed  as  essential  to  cost-effective  VV&A  are  sufficient  (in  a  formal  logical  sense)  to 
guarantee  the  cost-effectiveness  of  a  VV&A  process.  There  may  be  other  attributes  that 
contribute  to  cost-effectiveness,  or  it  could  be  that  other,  unconsidered,  factors  mitigate  against 
cost-effectiveness  even  when  the  features  considered  here  are  present.  On  the  basis  of  their  near- 
ubiquitous  presence  in  VV&A  processes  that  were  independently  designed  for  different  M&S 
types  and  applications,  however,  it  seems  reasonable  to  conclude  that  these  attributes  are,  at 
least,  "necessary"  in  the  logical  sense.  The  most  prominent,  well-accepted  and  widely  used 
VV&A  processes  just  seem  to  possess  these  features  in  some  form  or  another.  As  a  result,  it 
seems  reasonable  to  consider  these  features  normative  for  purposes  of  developing  or  evaluating 
VV&A  processes  for  new  M&S  types  or  classes  of  applications.  Moreover,  it  is  reasonable  to 
propose  these  features  as  the  ones  for  which  the  DoD  should  provide  some  means  of  centralized 
support,  to  encourage  their  development  for,  and  consistent  application  to,  the  many  types  of 
M&S  and  M&S  applications  which  come  under  the  purview  of  DoD  M&S  interests.  In  the 
interim,  the  features  proposed  here  can  be  used  to  evaluate  or  modify  existing  VV&A  processes 
by  using  them  to  identify  current  process  features  where  the  most  improvement  in  cost- 
effectiveness  can  be  made. 
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